A Sentence-Level Hierarchical BERT Model for Document Classification with Limited Labelled Data

The emergence of transformer models like BERT means that deep learning language can achieve reasonably good performance in document classification with few labelled instances. However, there is a lack evidence for the utility applying BERT-like on long few-shot scenarios. This paper introduces long-text-specific model—the Hierarchical Model (HBM)—that learns sentence-level features and works well Evaluation experiments demonstrate HBM can, only 50 to 200 instances, higher than existing state-of-the-art methods, especially when documents are long. Also, as an extra benefit HBM, salient sentences identified by useful explanations classifications. A user study demonstrates highlighting these effective way speed up annotation required interactive machine approaches active learning.